ARROW-5317: [Rust] [Parquet] impl IntoIterator for SerializedFileReader#4323
ARROW-5317: [Rust] [Parquet] impl IntoIterator for SerializedFileReader#4323FabioBatSilva wants to merge 3 commits intoapache:masterfrom
Conversation
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
Thanks @FabioBatSilva . It seems the newly introduced struct shares fairly amount of code with the existing RowIter. Instead, can we define a Either type like following:
enum Either<'a> {
Left(&'a FileReader),
Right(Box<FileReader>),
}And then use this in RowIter<'a>:
file_reader: Option<Either<'a>>,Then we just need to add one more method in RowIter that is similar to from_file but takes a Box<FileReader> instead.
Let me know what you think.
|
@sunchao I like the But I don't know if we can impl impl IntoIterator for SerializedFileReader<File> {
type Item = Row;
type IntoIter = RowIter;
//^^^^^^^ expected lifetime parameter
fn into_iter(self) -> Self::IntoIter {
RowIter::from_file_reader(Box::new(self))
}
}Perhaps we should extract the common logic from |
|
Do you still need let it = vec.iter()
.map(|p| {
File::open(p).unwrap()
})
.map(|f| {
SerializedFileReader::new(f).unwrap()
})
.flat_map(|reader| -> RowIter {
RowIter::from_file_into(None, reader).unwrap()
}) |
|
|
FabioBatSilva
force-pushed
the
reader-into-iter
branch
from
23c69ca to
7817ee7
Compare
3 participants
This patch adds a row iterator that takes ownership of the reader
and implements IntoIterator for SerializedFileReader
See :
#4301, https://issues.apache.org/jira/browse/ARROW-5317